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I INTRODUCTION 


As the health sector has passed from its brief 
tenure as first claimant on public funds to a position 
of having to compete with alternative uses of resources, 
the requirement that health administrators justify their 
requests for new programs and defend their present 
activities has grown. This in turn has led to demands 
for consistent, comparable and comprehensive defenses 
and justifications; the name given to this task has been 
"program evaluation". Increasingly health administra- 
tors are directed to evaluate their on-going programs or 
to commit themselves to an evaluation of new under- 
takings. Program managers, however, are not without 
misgivings about undertaking evaluation of their 
programs and they are frequently in doubt about how to 
undertake their evaluation. 


The Sub-Committee on Quality of Care and Research 
has *£ound fitsel£ yin this positions «Faced .incb973ewith 
questions about the value of cytology screening pro- 
grams, it commissioned a review of the state of the art 
of health program evaluation. A discussion paper* was 
favourably received by the Sub-Committee. It recom- 
mended that a Working Party on Program Evaluation be 
established to develop a document on the present state 
of the art of health program evaluation and to recommend 
future courses of action. (See Appendix II for the 
Working Party's Terms of Reference.) 


This volume is the report of the Working Party. It 
begins with an explanation of how the Working Party pro- 
ceeded with its task. It then summarizes the findings 
and recommendations of the Working Party, and concludes 
with several topical examples of deficiencies in the 
current practice of program evaluation. A comprehensive 
bibliography is presented as Appendix III. 
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*The Evaluation of Health Programs with Particular 
Reference to Screening Programs, David L. Martin, 
Health Insurance Directorate, Health Programs Branch, 
Department of National Health and Welfare, Ottawa, 
September 19173. 
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The second volume of the report is a Primer on 
Health Program Evaluation. It provides a detailed meth- 
odology for the execution of program evaluation with 
relevant references cited. 


The Working Party has generally recommended that 
the resources devoted to program evaluation be signifi- 
cantly increased. At the same time, however, it has 
recognized that there are many Structural features and 
incentive patterns in health care delivery which dis- 
courage evaluation. Although such evaluation is a 
currently fashionable topic of interest, the minimal 
development of genuine evaluative efforts found by the 
Working Party may reflect an absence of demand for such 
services by program administrators and managers. Like 
the weather, everyone talks about evaluation but nobody 
does anything about it. The Working Party were aware 
that there may be good reasons not to evaluate. 


If the current lack of evaluative effort is in fact 
due to absence of demand rather than unavailability of 
capable personnel and resources, a policy of expanding 
such resources would be futile. The Working Party has 
taken the view, however, that a genuine demand does 
exist for greater accountability, cost-effectiveness, 
and cost control in health care, if not by health care 
administrators then by their political masters. Under- 
lying all its work has been the presumption that such 
improved efficiency can come about only through improved 
program evaluation, particularly of existing programs. 
Even if one adopts a more pessimistic or "red-necked" 
view that rational analysis or control of health care 
delivery is impossible and that the only feasible 
strategy is one of pegging total health expenditures to 
a fixed percentage of G.N.P., come what may, the problem 
of allocation of total health resources so determined 
would still remain. Moreover, the percentage of G.N.P. 
used aS a peg, while in the short run perhaps based on 
last year's experience, must over longer periods be 
adjusted according to some evaluative criteria. One 
cannot avoid evaluation. One can, however, choose 
whether Of not to doll expriclelyy, eatcLonally, ana 
openly, or by random guesses. Rightly or wrongly, the 
Working Party puts greater trust in the former process. 


mA 
If THE PROCEEDINGS AND STUDIES OF THE WORKING GROUP 


Between June 1974, and June 1976, the Working Group 
met five times, with additional meetings held between 
those members of the Working Group responsible for 
individual sections of the final report. The other 
activities of the Working Group, including those carried 
out for it, are outlined briefly below: 


1. The production of an up-dated bibliography on 
program evaluation (Appendix III). 


2. A survey to determine from the provincial 
authorities their on-going and recently 
completed program evaluation projects. 


3. A review of National Health Grant financed 
evaluation projects (identified by specific key 
words in the NHG project listings). 


4. Presentations received by the Working Group 
from persons involved in or vitally interested 
in health program evaluation activities. 


5. A review by the Working Group members of the 
Health Care Evaluation Seminar manual. 


6. Site visits to three provinces in order to 
elaborate on the survey noted above. 


7. The preparation of position papers related to 
sections of this final report. 


8. The preparation of "A Primer on Health Program 
Evaluation", a manual to accompany this report. 


Early in its deliberations, the Working Party 
decided to ascertain the "State of the art" of program 
evaluation in the Canadian health care system. This 
background information was required to help identify 
problems, and to provide a context for our recommenda- 
tions. The first approach to developing this review was 
a survey of all Provanoualy Ministries "of “Heaith: 
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The information provided by the responding 
Ministries was disappointing. While the mailed survey 
elicited evidence of interesting projects, in the 
opinion of the Working Group it neither mirrored the 
"state of the art" nor adequately indicated whether 
program evaluation activities met Ministry needs. The 
Working Party decided to augment the information 
gathered by this survey by site visits to three 
provinces. 


A visiting team was formed consisting of members 
who had first-hand experience with evaluation activity 
in two of the provincial ministries of health and in the 
Department of National Health and Welfare. This team 
spent two days in each of three additional provincial 
ministries interviewing Senior civil servants. The 
interviewers used a Ssemi-Structured approach to allow 
the participants to express their views on program 
evaluation in their own terms. The participants were so 
candid and comprehensive that the interviewers rarely 
used an agenda to stimulate discussion or to orient the 
interviews. 


After san -initialssynthesis of .its,findings,,,the 
visiting team checked them with three additional prov- 
inces for consistency. 


The findings could not be presented as the results 
of a rigorous study. Nevertheless, the visiting team 
felt that in aggregate they represented the major prob- 
lems and issues in health care program evaluation in 
Canada today. 


The Working Group also discussed, during its delib- 
erations, the role of the Program Evaluation Division 
within the Program Development and Evaluation Direc- 
torate and the concerns of the Canadian Hospital 
Association. . 


In summary, the results of the Working Group's 
review of the Discussion Paper appear as Volume II of 
this report: aq"Health+«~Programsakvaluatilon.>As Primer! .. 
The examples of its application appear as section VI of 
this report volume. Its Summary statements and 
recommendations as to how health program evaluation can 
be improved follow immediately. 
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CONCLUSIONS OF THE WORKING GROUP ON PROGRAM 


VALU N 


There is a need for universally recognized def- 
initions of the terms "program" and "program 
evaluation". 


The definition of "program evaluation" should 
reflect its role as an essential component of the 
management process. 


The development of new health care technology and 
Knowledge, coupled with an economic climate of 

restraint, has created the demand for and current 
popularity of program evaluation. However, due to 


an insufficient number of trained people, diffusion 


of the program evaluation methodology has not kept 
pace with the needs. 


The current efforts in program evaluation are very 
limited: 


4.1 Frequently, no program evaluation is per- 
formed. Evaluated programs represent only a 
limited proportion of governmental budgets. 


4.2 Informal or imprecise enquiries are common. 


Frequently enquiries are limited to restricted 


aspects of the program. 


4.3 Insufficient funds and personnel have been 
allocated to program evaluation efforts. 


4.4 Program evaluation is often not built into the 


planning systems. In some instances, the 
Programming, Planning and Budgeting System 


procedures (PPBS) are superimposed on already- 
operative systems, rather than replacing them. 
Therefore, program evaluation becomes resented 


as a further imposition on management's 
restricted time. 


4.5 Cross-program evaluation appears to be virtu- 
ally non-existent. 
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There are powerful disincentives to doing program 
evaluation, including: 


5.1 the tendency in bureaucracies to reward pro- 
gram managers in proportion to the size of 
their expenditures. 


5.2 the deep personal involvement and investment 
by program managers in the future of their 
programs. 


5.3 a general resistance to change and, therefore, 
resistance to evaluation. 


There is an unequal (and perhaps unfair) scrutiny 
of new programs. While new programs may be easier 
to stop, existing programs have a much greater im- 
pact on total government spending. 


Present evaluation efforts appear to be unsophisti- 
cated. 


7.1 There is little information dissemination 
between program evaluators. 


7.2 Much redundant development of techniques and, 
conversely, little exploration of new concep- 
tual areas takes place. 


7.3 There is generally little understanding of the 
relative costs and benefits of information for 
decision-making. . 


Techniques of measuring costs, outputs and outcomes 
are relatively undeveloped. 


The allocation of resources for the acquisition, 
processing and dissemination of data describing the 
performance of health care programs needs 
re-thinking. 


9.1 Substantial amounts are spent on large 
computer and non-computer based information 
systems. Unfortunately these systems tend to 
be oriented primarily to business operations 
such as claims processing. 


LO. 
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9.2 These business systems consume an inordinate 
Share of the resources allocated for data 
processing, leaving few resources for infor- 
mation systems oriented solely to evaluation. 


9.3 Much of the data produced by these business 
systems could be of value, but there 1s a4 
reluctance to devote resources to transform 
Lois. data into userul information. 


9.4 There is an even Stronger resistance to add 
data items to these business systems which 
evaluators would find useful but are of no 
value to the business operations. 


9.5 There is little sharing of information and 
information systems between provinces. This 
makes comparison of programs -diificult. JlGwis 
also wasteful. 


There are insufficient numbers of persons trained 
in health program evaluation techniques to act in a 
staff capacity to health program managers. Few 
program managers are sufficiently conversant with 
program evaluation roles and techniques to under- 
Stand the contribution that program evaluation can 
make to their program management decisions. 


A key consideration in the present inability of 
program evaluators to fulfill the needs of program 
management lies in the lack of a coordinated and 
cohesive association of persons Knowledgeable and 
concerned about program evaluation. As a result: 


11.1 There are few effective mechanisms for the 
sharing cf technology and results among 
program evaluation experts or for. the 
dissemination of information on program 
evaluation resources. 


11.2 Centres of consultative expertise have not 
developed to the point where they could 
achieve their maximum potential contribution. 


11.3 Effective mechanisms to determine program 
evaluators' data base needs particularly for 
interprogram comparability are lacking. 


ssi es pe 


A : = mos i : 


a Ps : my 
HG ay Stat ais 4 fo 
| > at) Aducks prec avA, 
top 'Pedy re ae 2 “i 4 


30 RSE, 
i ai 


” : , | iy 
i : ( = : | , 
“a ink, eh, Rodaeelle aielaain ale Aeris 
> ine mT rngnate.” ees Pre x’ 
rs i, Gs tee 7 pe: 


Ja 


Pe 
7 
a. | 


IV 


1. 


IT IS THEREFORE RECOMMENDED THAT: 


The following definitions be referred to the 
Canadian Hospital Association and other appropriate 
Organizations for incorporation into their proceed- 
ings or publications, particularly their terminol- 
ogy manuals: 


1.1 The term "program" be recognized as a grouping 
of resources (including persons, funds, 
equipment and supplies) performing activities, 
all of which fall within definable boundaries 
enabling the scope of the program and its acti- 
vities to be measured, and achieving outputs 
(Or ultimately outcomes) aimed at Paltiiiang a 
definable set of objectives. 


1.2 "Program evaluation" be similarly defined as 
the measurement of the program's outputs or 
results and their comparison to Standards or 
to the resources made available to the 
program, for the purpose of defining policy, 
resources or objectives changes and, hence, 
altering the management of the program. 


Governmental and other agencies which are seriously 
concerned with improving the efficiency and 
effectiveness of health care delivery would be well 
advised to upgrade program evaluation resources 
generally, to bring the methodology Knowledge and 
resources in the field into alignment with current 
demands for program evaluation. 


Each program be required to include evaluation 
activities as a specific budget item, and program 
managers be held accountable for the agreed-upon 
performance of program evaluation activities. In 
the aggregate one-half of one per cent of the total 
health care budget should be set aside for program 
evaluation purposes. The proportion appropriate to 
each program, however, will possibly be greater for 
smaller or newer programs, depending upon the 
potential objectives of the specific evaluation. 
Furthermore, when budgeting tradeoffs become 
necessary between service delivery and evaluation 
‘of those delivery programs, the tradeoffs should * 
not be resolved in favour of service delivery. 
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At least equal priority be given evaluation acuLy— 
ities for existing programs as is required for 
proposed or new programs. 


Governmental and other authorities continue to 
encourage educational programs in program evalua- 
tion;s including: 


ae relevant educational programs at the 
university post-graduate level to prepare 
program evaluation experts, with appropriate 
financial support (in part through the 
National Health Scholarship Program) 


b. health care evaluation seminars for program 
Managers to increase awareness of the voten- 
tial uses and requirements for program evaluation; 


Cc. mechanisms for interchanges between academic, 
government and institutional environments 
early in the careers of those involved. 
Specifically, National Health Student 
Fellowships and Career Development Awards 
should be tenable at non-academic institu- 
CLons. 


Existing data systems be examined closely with a 
view to meeting the needs of program evaluators; 
specifically, since a very large volume of re- 
sources, probably of the order of $100 million in 
the case of fee-for-service medical insurance 
plans, is currently being spent on administering 
existing plans, priority should be given to 
redirecting a Significant proportion of these 
resources to the development of new management 
information systems which will permit evaluation of 
existing programs. 


Program evaluation data requirements be specifi- 
cally provided for in the plans for new programs. 


Program evaluators be included in those committees 
Or commissions where alterations to data bases are 
being contemplated. 


ae 


ou The Federal Government establish: 


a clearing house or information centre to 
receive and appropriately distribute informa- 
CLONT Cll: 


- program evaluation activities, particularly 
completed studies; 


- new or revised methods, new applications, 
Gata, etc.; 


- available evaluation consultants and their 
particular skills or expertise; 


- research activities funded from federal, 
provincial or other sources; 


10. The Federal Government make available to provincial 
governments $10 million (1977 dollars) to finance 
regional centres of program evaluation consultative 
expertise. 
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DISCUSSION 


There is a Need for a Universally Recognized 


Definition of the Term "Program". 


This problem arose early in the Working Group's 
discussions and, in fact, was also faced by the 
SubCommittee on Quality of Care and Research. The 
definition had to be sufficiently general to 
encompass an already-wide diversity of program 
types. It also had to be Specific enough so that 
the boundaries of the program could be defined and 
measured. 


It is important that the boundaries be definable 
and, aS will be outlined later in this report, 
extremely important that the objectives of the 
program be explicitly defined in obServable and 
measurable terms. The many possible boundary 
definitions for resources, activities, consumers or 
results are outlined in Volume II of this report 
("Health Program Evaluation - A Primer"). 


There is a Need for a Universally Recognized 


(sl Mabe hii AES eli eae Meth herb eal oP Sta mented) = Bode a 
Understanding of the Term "Program Evaluation". 


There is no universally recognized definition of 
program evaluation, recognizing that it can be 
either formal or informal in nature. 


The Working Group was aware that its definition 
would not include the evaluation of the objectives 
themselves. In most cases, a program constitutes 
part of a larger program or administration, which 
in turn’ sets the objectiives: not. only for the’ pro- 
gram under consideration, but for other programs as 
well. For example, if an agency encompassed three 
programs it would establish the objectives and 
resource allocations for these programs. This 
constitutes, in effect, the “output” of that 
agency. Thus, the sum total impact of the agency's 
programs would be evaluated as its total "program". 


ia 


The Working Group observed that studies which are 
not specifically oriented to contributing to better 
management decisions should be excluded from the 
scope of the Working Group's concern. On the other 
hand, feasibility studies which simulate potential 
programs, pilot studies, and similar ventures fall 
within the terms of this definition. 


The Working Group concluded that evaluation should 
be recognized as an integral part of management 
activities. While program evaluation might be 
envisaged as occurring during or following the 
completion of a program, planning for evaluation 
must be undertaken at the program's inception. 

The program plan developed by management must in- 
clude evaluation, and evaluation in turn will 
contribute to the managers' knowledge for program 
planning purposes. 


All program managers must recognize that they are 
program evaluators.. However, when the conceptual 
or technical problems of program evaluation become 
complex (as are the examples given in this volume 
and in Volume II), the program manager should feel 
free to call upon persons specifically trained to 
handle these complexities. 


The Development and Diffusion of Technology, 
Knowledge and Kesources have not kept pace with the 


Demand Created by the Needs and Current Popularity 


of Program Evaluation. 


Many factors have contributed to the current 
»vogue” for .program evaluation, including but’ not 
limited to: 


(a) the development of new administrative tech- 


niques, such as the programming, planning and 
budgeting system (P.P.B.S.); 


(b) the current concern about financial re- 
straints, and the need to know the value 
received for every dollar ,Spent in health 
programs; 
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(c) the development of new and more sophisticated 
health programs aimed at solving complex 
problems, with the result that definitions of 
resources required and results achieved become 
unclear; and 


(d) the increasing complexity of the health field 
generally, entailing the advent of new profes- 
Sions and techniques and resulting in greater 
dispersal or differentiation of the health 
care dollar. 


Virtually all government agencies and research 
funding sources now require that formal evaluation 
be built into new proposals. This has built new 
demands into the health care field for which it has 
not been prepared. Not only are there too few 
persons trained to undertake complex program evalu- 
ations, but program managers also do not understand 
the need for and contribution of program evalua- 
tion. Program evaluation is frequently included in 
new proposals only to satisfy the demands of finan- 
cial authorities. 


Frequently, the few available program evaluation 
experts also face data insufficient to their needs 
and technical problems for which approaches may have 
already been developed. Knowledge of these 
approaches may not be available as they are simply 
not publicized. What is generally required, 
therefore, iS a commitment on the part of 

government authorities and other health agencies to 
overcome them through effective communication 
channels. 


The Amount of Formal Program Evaluation is Very 


Limited. 


This statement involves a number of dimensions. 
First, looking at ‘the number’ of" persons on “a“‘fuli= 
time equivalent basis devoted to program evaluation 
efforts, the Working Group felt that 200 persons is 
a generous estimate for all of Canada: one quarter 
with the federal and provincial governments, one 
quarter in academic institutions, and about one 
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half in hospitals, other health research and deliv- 
ery organizations, and hospital and health assocla- 
tions. ~Inhis, estimate would reflect.a cost,of, vat 
most, $10 million in direct and indirect costs. 


The totalevannual cost.of healthidelivery os). EL 
billion. A common recommendation is that one half 
of one percent of program costs should be devoted 
tovevaluation ertortss., Inereftore,. oo million, 
rather than the present $10 million, would be a 
more appropriate investment. In order to catch up 
to this) criterion in 10 years; with present’ Infla- 
tion rates, an annual rate of growth of 20% in 
program evaluation budgets would be required. 


As a further indication of the limited scope of 
formal program evaluation, a survey was undertaken 
of the federal government, which included five 
departments, National Health and Welfare, Manpower 
and Immigration, Secretary of State, Indian and 
Northern Affairs, and the Department of the 
Solicitor General.* Within these departments, only 
38 studies were identified as having been com- 
pleted, three of which were in National Health and 
Welfare. The value of the programs concerned, 
however, was only $156 million, compared to a total 
value of almost $10 billion for the federal social 
welfare programs carried out by these five depart- 
ments. 


In its review of National Health Grant Supported 
projects, the Working Group found very few exempla- 
ry projects in terms of general applicability or 
methodology. Evaluation criteria often were not 
Stated, and relatively few projects could be 
generalized beyond the target group or sample 
population, the special health field involved, or 
the political boundaries described within the 
project. Most dealt with the effects of instru- 
ments or interventions, and few dealt with the 
benefits involved. Very few dealt with the effects 
of such interventions on the health system. 
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*Federal Level Evaluation, Leonard Rutman and 
Dick de Jong, Carleton University Graphic 
Services, Ottawa, 1976. 


ape 


The Working Group concluded that priority should be 
given to projects where the field can be general- 
ized and where effects on the health system are 
taken into account. Evaluators should be required 
to state their objectives, the expected outcomes 
and the scope or limitations of their projects. 


It is recognized, of course, that informal evalua- 
tion takes place. Frequently, these informal 
evaluations are undertaken because of the personal 
interest of a particular program manager. In many 
cases, they appear to be biased and self-serving 
because the program manager has a job and an 
Organization to protect. It is also noted that 
much of those funds that might be devoted to pro- 
gram evaluation from a long range perspective are 
devoted instead to short range "policing" evalua- 
tions, particularly in claims processing. 


A further measure of this impression was noted in 
the study referred to earlier. Their test for 
effectiveness of evaluative studies assumed three 
pre-conditions: 


tp A clearly articulated program; 
2% A clearly specified goal(s) and/or effect(s); 


a3 A rationale that linked the program to the 
goals and/or effects. 


None of 22 evaluations analysed in the above-noted 
study met all of these conditions. > In 

many cases the evaluators focused on effects that 
the programs were not designed to accomplish (i.e. 
many government programs are started without 
adequate consideration of the objectives of the 
program). 


In a number of instances, both provincial and fed- 
eral, the introduction’ of the P.P.B.S. ‘system or 
similar systems has resulted in some program 
evaluation activity. However, as they are present- 
ly conceived, these systems appear to be counter- 
productive. Many managers view the systems as more 
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bureaucratic paperwork to be circumvented, accom- 
modated or subverted. Rather than supplanting or 
replacing existing planning and operating systems, 
the systems have been superimposed and have not 
been accepted as a new modus operandi for health 
care program and planning decisions at the 
operating levels. 


The visiting team did find a substantial interest 
by program managers in rationalizing the system and 
evaluating their programs. Certainly, frustration 
with existing planning and decision processes was 
universal. 


This frustration was evident at all levels, politi- 
cian, Senior civil servant and program manager. 
This dissatisfaction with the present system and 
interest in evaluation can be turned to the advan- 
tage of the health care system, assuming the stated 
interest is not lip-service. In the opinion of 
those visiting the provinces, the interest was 
genuine. 


Within the Department of National Health and 
Welfare itself, all branches have one or more eval- 
uation units, but only one welfare program has been 
evaluated. Three evaluation studies have been 
completed in health (none by an evaluation group). 
Only one study, an evaluation of the Health 
Resources Fund, is currently underway. 


This should not be taken to mean that evaluation is 
absent from the federal public service. An opera- 
tional performance measurement system is being 
introduced, but at the present time it is primarily 
concerned only with operational efficiency. Higher 
proxy level analyses (such as operational effec- 
tiveness, program effectiveness or contribution to 
well-being) are not being undertaken. On the other 
hand, total reviews of government programs are 
being undertaken, with a review of welfare legisla- 
tion completed recently ("The Orange Paper") and 
health care legislation (particularly insurance 
programs) currently under review. To some extent, 
however, these cannot be termed formal program 
evaluation routines. 


SF 
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There are Powerful Disincentives to doing Program 


Evaluation 


The traditional incentive for the program manager 
is bigness, defined by expenditure levels, the 
number of people employed, particularly profes- 
Sionals, etc. The general rule is: the larger 
the organization, the more resources consumed, the 
greater the prestige and income for the manager. 


In many of the jurisdictions the visiting teams 
examined, bigness appeared to be the most important 
measure of worth. There were few analogues to the 
"profit line" of private industry operating in the 
Canadian health care system. Consequently, most 
evaluation activity is oriented to providing infor- 
mation to the program manager to allow him or her 
to demonstrate that the program is working well 
with existing expenditures and could do better with 
additional expenditures. 


A hospital administrator described the 

division of budgets as the distribution of rewards. 
Department managers typically look for increases in 
department budgets. They regard decreases or even 
"holding the line" as a punishment. Rarely would a 
department head look upon a reduction in his 
departmental budget as a demonstration of and 
reward for good efforts. 


In addition, program managers have a personal 
investment and interest in the future of their 
programs. The visiting team frequently observed 
that program managers appeared to be zealots for 
their own programs. Few evaluated their programs 
with test hypotheses that questioned the basic 
worth of their programs, or looked for substitutes 
that would replace them. 


Finally, the implementation of program evaluation 
concepts faces strong resistance to change. The 
most secure civil servant will view with some 
trepidation an inquiry into the operations of his 
department or program, which may bring about 
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significant changes or even the discontinuance of 
his program. Even when he himself is clearly 
convinced of the need for his program, the arrival 
of the evaluator is never greeted with eager 
anticipation. 


There is an Unequal and Perhaps Unfair Scrutiny of 
New Programs. 


A number of provinces have introduced Program 
Planning and Budgeting Systems on a phased basis. 
Usually only new programs are required to conform 
to the additional scrutiny demanded by these 
systems, while existing programs continue to be 
funded and managed in the traditional way. 


In those provinces that have not embarked on 
P.P.B.S., most have two budgets, usually called 
Budget A and Budget B. The federal government, 
with P.P.B.S., also has "A" and "B" budgets. The 
new programs (Budget B) must be scrutinized and 
justified far more extensively than the ongoing 
programs of Budget A. 


Improved evaluation must start somewhere. New pro- 
grams, before they are operational, are much easier 
to stop politically than existing programs, but 
they usually represent a very small proportion of 
the budget. In the opinion of the Working Group, 
the primary consideration in selecting programs for 
a phased-in program evaluation process should be 
the size of budget. Other considerations include 
efficacy, impact and cost-effectiveness. 


The Present Evaluation Efforts appear to be 


Unsophisticate 


It is recognized that increased sophistication is 
only relevant if the additional information gained 
is worth the price. However, frequently the costs 
of the additional sophistication are unknown, and 
the data it requires renders it virtually impos- 
Sible. 


The visiting teams found few examples of inter- 
provincial trading of information systems, data or 
program evaluation results. There is much 
"reinventing of the wheel". In each provincial 
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health department, program evaluation routines were 
being designed, many of which have been devised, 
tried, and frequently found wanting elsewhere. 


There appeared to be little exploration of new con- 
ceptual areas for program evaluation. The majority 
of efforts appeared to be devoted to under- 
developed cost-benefit studies and data collection 
exercises, where it was assumed that the data would 
somehow yield the required answers. While a few 
program managers belied the above observations, 
they were exceptions to the rule. 


Finally, existing management systems appear to be 
deficient in that they do not deal with cross- 
program evaluation. Those visiting the provinces 
Saw no evidence of decision making that dealt with 
trade-offs in budget allocation between two or more 
programs. 


There appears to be little understanding of the 
relative costs and benefits of information for 
decision-making. As a result, many decisions tend 
to be made intuitively, frequently with poor 
results. These obServations are more fully discus- 
sed in Section 8 below, and in Appendix C of Volume 
Zioibsthrs report ; 


A Major Obstacle to Health Program Evaluation is 


the Relatively Undeveloped Nature of the Techniques 
of Measuring Inputs, Outputs, and Outcomes. 


When the Working Group members surveying the prov- 
inces inquired about measures of health care 
effectiveness, they were frequently confronted with 


the old saying: "After all, how can you really 


measure sthe valuasof aslhifesesaveds" 


While this problem is obviously difficult, program 
evaluation can proceed. A single measure that 
linked or allowed comparative evaluation of output 
would be invaluable. However, it is not a pre- 
requisite to program evaluation activity. If we 
begin with the specification of program objectives, 
the measurement of the attainment of these 
objectives, including accurate measurement of all 
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costs incurred, the learning experiences would 
undoubtedly lead to improvements and refinements. 


As noted above, the visiting members of the Working 
Group found a paucity of effort devoted to develop- 
ing new measures of program performance. It was 
felt that the major reason for this was not the 
difficulty, but the lack of management systems that 
encouraged their development and use. 


In those evaluation activities that were under- 
taken, strategic and technical errors were made 
which led to invalid conclusions. The profound 
impact of these errors has led this Working Group 
to devote a special section of this report to 
examples of them. (See Section VI). 


The Allocation of Resources for the Acquisition, 


Processing and Dissemination of Data Describing the 
Performance of Health Care Programs Needs 
Re-thinking. 


The first problem relates to data availability. A 
common complaint of the program managers inter- 
viewed was the amount of potentially valuable data 
which has not been adequately synthesized to meet 
their needs. Substantial efforts would be required 
to process the data. 


Second, provincial and federal health program 
managers stated that the existing information 
systems rarely met their requirements. The 
systems, while processing large amounts of data, 
could only respond very slowly when required to 
process existing data in new ways to answer new 
questions. Therefore, the data available would 
frequently be inappropriate, invalid; unreliable; 
untimely and inconsistent. 


Data collection authorities are even less enthusi-= 
astic. about collecting inew, data that do not readily 
fLeeIntee. or (serve (theif cexisting systeme. dRathes 
than attempting to devise new data sources and 
collection routines, incremental changes will only 
be made, if possible, to existing statistical 


Surveys, (e.g., the Annual Returns for Hospitals, 
H5—Evand: HS=2). 


10. 
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There are Insufficient Numbers of Persons Trained 
in Program Evaluation Methods to Support the 


Existing Managers. 


This conclusion has been reached by many in the 
health care system: 


im) It was stated many times to the visiting team. 


2) It has been stated by representatives from 
existing evaluation groups who have been 
called upon to assist numerous health 
departments and health agencies to evaluate 
their own programs. 


3) The relatively few studies undertaken in this 
field (reflected in Section 4 above) and the 
Crucial errors that are being made (as re- 
flected in Section VI) point to a crucial need 
for effective analytical assistance to program 
managers to aid them in making key decisions. 


It has not been possible to develop anything more 
than an eStimate of the resources available or to 
give any idea as to what the potential need would 
be. 


Sources of consultative expertise, particularly 
health administration programs, are being hard 
pressed to provide consultative or even personnel 
assistance in evaluation of various government or 
health institution.programs. Demand for this ~ 
service outstrips the supply in terms of available 
faculty hours. There is also no suitable or 
appropriate financial base to support such 
activities. Resources would have to be pulled away 
from teaching other research programs... 


Efforts to improve the program managers' knowledge 
of quantitative methods and program evaluation 
techniques are badly needed. There are numerous 
ways of doing this, including establishing exchange 
programs between health agencies and university 
health administration programs; establishing man- 
agement information seminars on program evaluation 
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or incorporating program evaluation into existing 
seminars, conferences and conventions for health 
care managers; and encouraging the development of 
papers in general health administration literature 
on the uses and effects of program evaluation . 
techniques. When program managers enhance their 
understanding of how program evaluators can assist 
them, the Working Group feels that the need for 
program evaluators will increase, and at the same 
time become more readily defined. 


One step in the education of program administrators 
is the development of a "primer" in health program 
evaluation. The Working Group has complied with 
the request by the Sub-Committee on Quality of Care 
and Research and presented it as Volume II of this 
report. This volume has attempted to define in 
relatively simple terms the role of and steps 
involved in health program evaluation. For those 
interested in studying this field, it has also 
provided an introduction to other literature on the 
subject and a bibliography. 


There is a Lack of a Coordinated and Cohesive 
Association Of Persons Knowledgeable and Concerned 
About Health Program Evaluation. 


It appeared obvious to the Working Group that pro- 
gram evaluators operate in isolation. There are no 
ready channels for the dissemination of information 
on program evaluations undertaken, their techniques 
Or results, new data bases established, or even the 
whereabouts of the persons working in the field. 

As a result, there is much wasted effort. 


An example of the waste is the former dissemination 
policy of the National Health Research and Develop- 
ment Grants (formerly Directorate of National 
Healen Grants). Until™ recently, no*public dis-— 
tribution was attempted, not even the publishing of 
abstracts. The reports submitted to that Director- 
ate were simply filed. It was virtually impossible 
for anyone outside of the research project group to 
learn of the techniques or findings. The Working 
Group understands that this situation will change, 
that previous projects will be made available 
through the Departmental Library, and that future 
project reports will be reproduced, where appro— 
priate, by the Directorate, 
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A clearing house for information on evaluation and 
evaluators, encompassing proposed projects, newly 
developed technical approaches, current evaluation 
research findings, the education and locations of 
evaluation experts, and newly published materials 
in the field would solve many of these problems. 

It could be based within the Department of National 
Health and Welfare and would help draw together 


professional health care evaluators at a reasonable 
COST. 


A periodic forum of program evaluators would be 
beneficial. New program evaluation technical 
approaches could be presented, questioned or 
deLtended in front of a ccritical ‘audience. pine 
forum would permit program evaluators to define 
areas where new techniques and data bases are 
required. It could also help avoid duplication in 
technique development, and assist in the definition 
of appropriate data bases (thus supplementing the 
efforts of the Interdepartmental Committee on 
Health Statistics). This could be supplemented by 
regional forums if appropriate. 


The forum would also prove a useful supplement to 
the existing Health Care Evaluation Seminars 
referred to previously. The two would fulfill 
complementary but distinctly different needs, the 
health care evaluation seminars providing intro- 
Guctory courses to the field and the annual forums 
providing the field with the necessary opportunity 
to develop itself appropriately. 
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EXAMPLES OF APPLICATIONS FOR HEALTH PROGRAM 
EVALUATION: CONCEPTUAL AND OTHER PROBLEM AREAS 


Summar y 


This section outlines some of the reasons why 
evaluation efforts have not been useful for 
decision making. The objectives pursued by many 
health programs are so broadly stated that it 
becomes impossible to understand their justifica- 
tion. Too often, the evaluation sequence of 
program preparation is neglected, both in terms of 
conceptualization and of operationalization. For 
this essential component of the administrative 
cycle, conventional wisdom is a poor substitute. 
In addition to the cardinal error of failing to 
specify in observable and measurable terms the 
explicit objectives set for a given program, this 
Section 1llustrates five other common fallures, - 


with brief examples. 


The first is the failure to state what is expected 
to occur if the program is not implemented. The 
effort of specification forces one to state explic- 
itly what is otherwise assumed implicitly. The 
validity of these assumptions can only be assessed 
in the open. (See the influenza vaccine program 
and home care program examples.) 


The second is the failure to specify the complete 
range of activities and inventory of resources 
involved in the realization of the program, (as in 
the transplant perfusion program and Influenza 
vaccination program examples.) 


The third is the failure to adopt a consistent 
standpoint for costing the resources used in 
programs.(See asbestos miners example.) 


The fourth is a composite mix of errors made by 
aggregating benefits over time and participating in 
the universal reluctance to discount: them to their 
present value. Any program, such as Home Care or 
Day Surgery, which requires changes in the capital 
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stock (e.g., fewer inpatient beds) has smaller 
initial benefits which grow over time. However, 
future benefits are less valuable than present 
ones. In other cases where benefits are recovered 
over longer time frames, (e.g., shared administra- 
tion programs), even more complex formulae are 
necessary. 


The laste failure 16) related to the costing, out of 
capital at a single point in time (expensing) 
rather than over time (depreciation and implicit 
interest cost). Even worse is to ignore capital 
cost entirely because it is in some other budget. 


Introduction 


The process and purposes of evaluation can be de- 
fined, diffused, and ramified without limit, but 
the essential idea is nevertheless very Simple. 
Evaluation requires the assignment of values, and 
value is a relative concept, in two major senses. 


First, all values are defined relative to an ex- 
plicit or implicit evaluator. There must be some 
individual or collective group, either real or 
hypothesized, with objectives such that activities 
or objects take on values relative to those objec- 
tives. Change the identity or objectives of that 
evaluator and the values change too. Second, all 
valuation is relative in that it presupposes (often 
implicitly) a choice between two or more options, 
courses of action, or states-of-the-world. Each is 
valued relative to the other. There is no objec- 
tive definiton of value, as a quality inherent in 
an object or a policy, which can be established by 
a process of "Scientific" evaluation. Some of the 
rational and analytic processes involved in eval- 
uative work may be similar to those used in 
scientific research; but scientific evaluation as 
Such 1S a contradiction in terms. 


These principles may be illustrated by an example 
from the real estate market. We often speak of ‘a 
particular house as worth so many dollars at a 
certain point in time; this appears to be an objec- 
tive statement of fact (which of course may be more 
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or less accurate) unrelated either to a specific 
evaluator or a specific alternative state-of-the- 
world. But in fact a professional appraiser, in 
making such an evaluation, is attempting to 
estimate what some buyer will pay for it. He may 
not know who that buyer will be, so the estimate 
must be based on what (presumably) similar buyers 
have paid for similar houses. When the house is 
sold, its value (or at least a lower bound estimate 
of its value) becomes known and supersedes the 
appraiser's estimate. The appraiser may judge that 
the house sold for more than it is worth, which 
means that he believes no other buyer would have 
paid as much. But unless he has reason to believe 
the buyer was misinformed about the house (or about 
other alternatives) this statement is irrelevant. 
The house is worth what the (informed) buyer will 
pay, to that buyer. Apparently objective market 
evaluations thus represent values defined relative 
to the preferences of an ill-specified group of 
market participants, whose subjective preferences 
have been aggregated in a specific way. 


This specification of standpoint becomes particu- 
larly critical in health care programs, where one 
can evaluate from the point of view of an institu- 
tion, or a sub-department, or a government, or the 
society as a whole. Outside evaluators tend to 
take the system - wide or societal viewpoint, 
implicitly treating the whole society as the buyers 
OfPyprogram output.” Thus ‘a~ (potentially) eoce— 
reducing home care program will be evaluated 
relative to inpatient facilities from the point of 
view of patients who will be served by the facili- 
ties and of taxpayers who will support them. The 
value of the program from the point of view of a 
local hospital, some of whose inpatient facilities 
will be (or should be - they usually are not) 
closed when the new service becomes available, will 
of course be very different, just as the seller of 
the house looks at the purchase price very dif- 
ferently from the buyer. One man's cost is another 
man"s benefit, thus they will tend to produce 
Gifferent answers to cost-benefit comparisons. ‘The 
peculiar feature of health care delivery is that 
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although evaluators tend to start from a societal 
viewpoint, operational control of programs by 
providers tends to. respond, to values. as; seen iby 
providers. Hence the common discrepancies between 
what is done and what evaluators judge should be 
done. It is as if sellers of houses were able to 
require buyers to purchase houses and to pay prices 
determined by sellers' ("expert") evaluations - one 
would not be surprised to see higher prices as a 
result. 


The second form of relativity arises because a 
house, iike a program, is not, valued in isolation. 
Thesstatement. tiiatslt 1s worth, Sx; means that ete 
appraiser believes that some representative buyer, 
Or some buyer who will arrive in the next month (or 
three months, or. 2...) would compate. two states -of— 
the-world, one in which he has $x in the bank and 
one in which he owns the house but no money, and 
finds the second state preferable or at least equal 
to the first. Alternatively the states may be 
ownership of house encumbered by $x mortgage 
against no house, no mortgage, or some intermediate 
State. Whatever the specification of these alter- 
natives, some such choice must be possible. It is 
easy to see that one who receives the house as a 
gift or inheritance will value this state-of-the- 
world differently from one who has simultaneously 
acquired a large, high interest mortgage. To 
evaluate the acquisition of the house from the 
point of view of the new owner it is critical to 
know what else iS going on. 


This principle generalizes to health care program 
evaluation. Major institutional changes, actual or 
projected, induce extenSive system responses and 
have extensive indirect as well as direct effects. 
One cannot evaluate their impact without identify- 
ing these responses, and without specifying, either 
explicitly of amplrcitly,. what, else 1s. going on. In 
general, the statement should be explicit. to,pro- 
tect “against the, likelihood, that. some.of. the 
implicitly assumed responses will turn out to be 
implausible. Thus, in evaluating a new project or 
PLOOGK al wie 1s. very inportant, to try, to. be, as 
complete and as concrete as possible in describing 
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the effect upon the relevant variables in the world 
with and without the program. It is thus possible 
to compare and evaluate their complete alternative 
states. 


It follows that evaluation has an inevitable hypo- 
thetical or "theoretical" component. Even if one 
evaluates an existing program, such evaluation must 
always be relative to some hypothetical world where 
the program did not exist. One may fail to specify 
the alternative explicitly and/or may specify it 
badly, but in the evaluative process an alternative 
(one at least) is always there by definition. 


Beyond these fundamental principles, there is much 
that one can say about evaluation. In particular, 
the complete and accurate evaluation of (or placing 
of appropriate relative values upon) the various 
components of the state-of-the-world alternatives 
is by no means an obvious process. However, it may 
be instructive to consider real-life examples of 
the sort of programs which tend to be evaluated and 
to show how certain types of baSic errors can be 
made in the evaluation process. AS a reSult very 
large biases can be built into the results (some- 
times deliberately). The point of these examples, 
which are cut-down versions of real life, is to 
illustrate certain problems and flag certain pit- 
falls. They should not be considered "how to" 
guides for doing the evaluations of the specific 
types of programs mentioned. 


Home Care Programs 


One of the selling points which has been used to 
justify the introduction and extension of home care 
(HC) programs is the capacity of such programs to 


-lower the rate of acute care hospital bed use and, 


thereby, moderate the high cost of hospital care. 

The same arguments are, of course, made for chronic 
and convalescent hospitalization, nursing homes, 

and alternative forms of institutionalization. The 
crudest form of "evaluation" supporting this claim 
provides a good example of inadequate or inaccurate 
specification of the net effects of the program, or 
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the state-of-the-world with and without home . 
care.l this is the popular technique of counting 
the number of person-days of care provided to home 
care enrollees, multiplying this number by the 
average per diem cost of hospital ‘care in’ the local 
area, and using the result as an estimate of the 
total hospital costs "Saved" by the program. 


For example, a home care program which had provided 
2000 person-days of care in 1975 might operate ina 
region or be based in a hospital where the per diem 
cost of hospital care was $80... "If programe costs 
were $15 per person-day in home care, an enthusias- 
tic program administrator might declare "savings" 
Of, S807x 2000 =75160,000, less *Si5 x Z000F= $307 000 
or $130,000 per year. Such "Savings" would appear 
to constitute a strong argument for expanding the 
home care program. 


Such an argument embodies strong and somewhat im- 
plausible assumptions about what would have 
happened in the absence of the program. It is 
assumed that all those persons in the HC Program 
would otherwise have been hospitalized, and for 
exactly the same number of days. Following this 
reasoning, the total inpatient-day count in the 
local hospital would have been 2000 higher in the 
absence of the HC Program. Furthermore, acute care 
hospital costs would have been proportionately 
higher. If the local area in 1975 actually had a 
patient-day count of 20,000 (which on a per diem of 
$80 implies a hospital budget of $80 x 20,000 = 
$1,600,000), then it is assumed that in-patient 
utilization would have been 10% higher and costs 
would also have been 10% higher. 


These assumptions are implausible for a number of 
rédsous.. Pilret, wrth respect to itil ization .oiteeis 
frequently observed that patients discharged from 
hospital to home care will spend longer in total 
Under -care than ni the HC "Program ‘hadinot tempeted,. 
The acute episode may be shorter, but the acute 
plus home episode is longer. 
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Second, patients may well be admitted to HC from 
the community without an acute care episode. It is 
possible, but by no means certain, that such cases 
represent avoided admissions which would in the 
absence of HC have become inpatients. Moreover, if 
the HC program restricts itself to admitting only 
patients discharged from acute care, it may find 
itself actually generating hospital use due to a 
tendency for patients to be admitted for a short 
time inorder to qualify for HC. Thus the: actual 
number of acute care days avoided by the HC program 
will, in general, be less than the number provided 
in the HC Program by an amount which will be 
difficult to determine but may be very 
significant.2 


Therefore, the implicit assumption that each HC day 
offsets one acute day iS inaccurate and overstates 
Savings, but the problem does not end there. If 
acute care days are avoided, they will not, in 
general; bes "full-cost": days. This ‘arises. for 
several reasons. First, the full cost per diem in- 
cludes a variety of costs incurred in the hospital 
for services other than inpatient care. Teaching 
hospitals are the most obvious example: if a sig- 
nificant part of a hospital's costs are incurred 
for medical education these costs will not corre- 
spond proportionately to the inpatient case load. 
The same is clearly true of the diagnostic depart- 
ments and clinics which function partly for out- 
patients. Moreover, the hospital's total costs 
include a large fixed cost component, independent 
of care load, which is determined by the scale of 
the establishment. Finally, the types of cases 
admitted to home care will be cases of minimal 
service intensity, either because the case is in 
the late acute stage when discharged or because 
extensive (and expensive) diagnostic or therapeutic 
intervention is not required. Therefore, the 
actual change in costs associated with transferring 
patients between home and acute care are likely to 
be "hotel" costs (room,. board, laundry, etc.), 
which will be much lower than the per diem of the 
hospital as a whole. But even these are only 
partly avoided. Some are merely transferred. 
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The "hotel" costs include food, but the HC patient 
also eatsesel The datferencermis. that inyacutemcare, 
the hospital budget meets these costs - in HC they 
are absorbed in the patient's budget. Food costs 
are y"saved" fromthe point, ofaviewaof theceréims 
bursing agency, but not from a social point of 
view. 


Hence, we may reformulate our example. Suppose the 
hospital with a per diem of $80 actually has 
"hotel" costs of $20, so that shortening the length 
of stay of a recuperating surgical patient only 
saves $20. Assuming that $5 of this is food costs 
and that 1000 of the HC person-days are avoided 
acute-care hospital days (the rest are additional 
days of care), then hospital costs saved are 1000 x 
$20 = $20,000. However, of this amount, $5000 
represents costs not saved by society, only trans- 
ferred from the hospital to the patient. The total 
CoStrof, the HCpPrognamy wer sth) Si5exeZ000gS 
$30,000, but a "Saving" of $160,000 has become a 
Nnetacostvot $307, 0000-615 ,.000 =" $157 000,45 pluste 
transter of S$5000-from hospital tto*patient* bud= 
gets. 


Of course the HC Program may still be worth while 
-- the extra 1000 HC days at $15 each may be con- 
Sidered a social benefit worth the price. But if 
the HC Program is an inSured benefit, one cannot be 
sure. This follows because the individual patient 
and/or physician will seek entry to HC (or more 
important, resist discharge) so long as the per- 
ceived benefit is positive. If any benefit is 
belteved, tos toddow irom. HC. if) ity ep it ree" seione 
CONTINUES ti. Bute uyis: nots Wfreelwed ity cost siySis5 
per day. Therefore, there is a possibility of 
overuse which suggests that, in general, the: extra 
days of care may not» all be “worth-while". 


And there is a final complication. Under current 
reimbursement modes for physicians and hospitals, 
there are strong economic incentives not to leave 
beds empty.m  Thus,4 the 000. days vof acute: Gare 
actually avoided by the HC Program may, by releas- 
ing beds, raise the admission rate for other and 
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bncelated, conditions; «This wild) not,occur, athe 
HC Program is introduced in a growing community as 
anpalternatave, to; building morewbeds,+ ons if) ib) dis 
introduced at the same time as withdrawal of facil- 
ities in a static community. But if the inpatient 
bed stock is not adjusted to reflect the new 
program, there is a strong possibility that other 
admissions may rise. If "hotel type" patient days 
are avoided but complete new admissions are accom- 
modated, hospital costs will actually rise in 
addition to the HC: costs. 


In general, a proper evaluation of the HC program 
must involve a calculation of total expected 
hospital utilization in the relevant region, in- 
cluding cases of HC type and others, with and 
without the HC program. This will have to be much 
more sophisticated than simply assuming a one-to- 
one relationship of acute to home care days, and 
should include information on a diagnosis specific 
basis, aS well as population and physician trends. 
If the program being evaluated is operating, the 
"no program" utilization estimate will be hypothe- 
Sized, but could be based on trend data on 
inpatient admissions by diagnosis, region, and 
physician. If a proposed new HC Program is being 
evaluated, projection of both cases will have to 
depend on examination of trends, and it will not be 
possible to look for changes in trends on or 
shortly after program introduction. 


Given two alternatives, with well-specified and 
justified utilization patterns for inpatient care, 
with and without HC, one must then reconstruct 
hypothetical budgets for the hospital(s) in the 
relevant region. These will reflect old and new 
utilization patterns, and ideally should be 
sensitive? to; differences in, diagnosticimix» of in 
patient load between the HC/no HC situation, as 
well as to differences in admission rate and length 
of stay. In constructing such hypothetical budgets 
account should be taken of the fact that many 
departmental expenditures in hospitals do not 
adjust proportionately to patient load. (AS an 
extreme example, the HC Program itself might be 
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part of, theshospital.: in thistcase,eare will have 
tor ibe: Vaken, that HC) costs “include any “services 
provided: by the hospital, at a fair ‘cost, and that 
such costs are excluded from the hospital budget.) 


Finally, one should compile an estimate of total 
hospital plus HC costs to compare with hospital 
costs) 1b ‘tbhererwere novHC. This) wilb*show the net 
cost of the HC Program (which would be negative if 
the HC Program really does have a significant 
impact on hospital use). This must be adjusted for 
changes in cost which are merely transferred from 
health care systems to patient or vice versa. 
Against this net cost may be set the change in 
health care provided, the evaluation of which will 
depend on the comparability of acute and home days. 
Three classes of persons may be identified: 


(5) those who are in home care who would not 
otherwise have been cared for, 


(2) those who are in home care who would otherwise 
have been in acute care, 


(3) those who are in acute care as a result of the 
freeing up of inpatient beds by the HC Program 
and who would otherwise have been treated on 
an ambulatory basis. 


The health benefits for group (2) are positive, for 
(lL) sand’ (3) the net benefit is less clear. sBut “it 
is the benefit to these groups which must be set 
against the net costs of the HC Program. 


Depending on the response of the acute care sector, 
these total net costs may be negative ("saved" 
hospital costs greater than direct HC Program 
COSES)— One posvelvern(panticularly as*utmuzaetion 
shifts toward more admissions and shorter stays) or 
Somewhere in between. There is no way to tell in 
advance what» the effect of the program will ibe. 
Moreover, the’ reconstruction of the hypothetical 
pattern of inpatient use and the corresponding 
budget for inpatient services is unavoidably un- 
certain and dependent on estimation and inspired 
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guesswork. What must be recognized and accepted is 
that such a process cannot be avoided -- a single 
and crude "evaluative" process such as initially 
described (per diem X HC person days) embodies such 
a hypothetical reconstruction implicitly (one-to- 
one offset of HC and acute days, proportional 
adjustment of hospital budget). Once made explic- 
it, such a simple reconstruction is seen to be 
inadequate, biased, and rather silly. But a more 
sophisticated reconstruction can be rather diffi- 
cule. 


Illustrations from an Organ Perfusion Program 


In general, the evaluative components associated 
with research project proposals, particularly in 
applications submitted for National Health and 
Welfare Research Grants, are either ill-defined or 
non-existent.” It. is commonly very difficult to 
criticize such proposals because of the impossi- 
bility of establishing the type of evaluation and 
HOW. Jt 1S LO"be Carried out. One particular 
proposal, however, from a hospital which proposed 
to establish and evaluate an organ perfusion and 
preservation program (an elaborate and highly 
sophisticated technique to preserve living tissues 
pending transplant), was sufficiently detailed to 
provide a clear picture of the procedures intended, 
thereby serving as a useful illustration of common 
flaws in evaluative efforts. It should be empha- 
sized that these flaws arise, to a large extent, 
from the nature of present hospital budgeting, 
Management, and reimbursement practices. No criti- 
cism of the hospital management involved is 
intended. | 


Incomplete Cost Statements 


Expected total costs of the organ perfusion and 
preservation program were reported as Salaries, 
remuneration and fringe benefits, drugs and sup- 
plies, and depreciation on major equipment. This 
breakdown obviously conformed to normal hospital 
accounting practice of not assigning overhead 
expenses to programs. Taken literally, however, it 
implied that the program would be carried out on 
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the hospital front lawn. No allowance was made for 
the cost of space or administrative overhead, 
either on a current basis or as an allocation of 
capital costs. 


Of <ourse;,16.1s probably, true hati .bor any one 
program no new building will be built or adminis-— 
trative staff hired. But the total expansion of 
hospital activities by such new programs will 
eventually lead to additional current expenses for 
administration and space. On average, such costs 
seem to add 30% - 50% to operating costs of pro- 
grams. They include: 


a) Current costs of administration. 


b) Current costs of space maintenance (heat, 
light, water, cleaning, security). 


c) Capital costs of buildings and equipment - 
both depreciation and pure interest cost. 


Envthisscontext, it should be noted that capital 
costs, both buildings and machinery, should include 
interest as well as depreciation. Certainly capi- 
tal wears out over a period of years, and the 
amount of wear in any year attributable to a pro- 
glam Sis saicosteot ‘that program::.But .ehe. capital 
used also-has an opportunity cost in addition to 
the wear factor - resources used to provide any 
piece of capital are then unavailable for other 
purposes. This cost is reflected in the market 
interest rate, which indicates the value of program 
GCapltal) puteto-some other, use... in..the.case. of. the 
SLgan perfusion, program, this factor is unlikebyeto 
be. large, but it should not be left out. 


The separation of capital and operating budgets in 
hospitals discourages such accounting - capital 
cOSES are. note charged: tothe hospi tal....,Buicthe 
tCruescost to the, province and, its: taxpayers, of, such 
a program certainly includes the foregone return 
that could have been earned on the capital used in 
that program, even if the best alternative use were 
toupay, off a small part) of .the, provincial. debtsesand 
Cerlecethe capital flow, back oto the: private. secuer. 
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It would be desirable if the existing budget frame- 
work could distinguish between budgetary and non- 
budgetary costs. At present, hospital budgets are 
incomplete cost statements, but costs are no less 
real for not being recorded. 


No Clear Specification of Alternative "States of 
the World" 


The proposal provided information on the mechanics 
of perfusion, together with a machine sketch. It 
did not, however, provide a description of what the 
program was expected to do as opposed to what 
perfusion will do. What was lacking was a clear 
outline of the expected pattern of work in a 
specific or hypothetical future year with and 
without the perfusion service. Evaluation is 
inherently comparative - we must compare 
(necessarily hypothetical) future "worlds", one 
including the service, the other without it. 


How many transplants would take place in each? The 
proposal indicated 15 without, 30 with, and that an 
estimated 30 perfusions would take place. MThis 
Suggested that all transplanted kidneys would be 
perfused. The program appeared both to expand ser- 
vice and to upgrade existing services. 


What would have happened to the 15 transplant 
patients in the absence of the program? Dialysis? 
Death? What is the long-run prognosis for the 
untreated patient, the dialysed patient, the 
transplant patient with and without a perfused 
organ? Estimates will be subjective and uncertain, 
and the evaluator will need the technical special- 
ist's “best guess" in these matters. In general, 
one should start with a kidney patient arising 
spontaneously in the population and should trace 
him through the alternative treatment paths through 
the expected range of outcomes. The role of 
perfusion in altering these outcomes can then be 
described. 
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Incomplete Specification of Program Evaluated 


Perfusiony preservation is only a part Cf the total 
pattern of treatment for kidney diseases, but this 
whole pattern is expected to be influenced by the 
availability of serftusion. «If perfustonschanges 
treatment patterns, we would be considering only a 
Smalls pogeton of the real. program. Thissreel 
program would be: introduction of perfusion result- 
ing in an increase in surgical activity, more 
transplants, perhaps less dialysis but perhaps more 
if all capacity continued to be fully used, and 
either more or less hospital care depending on 
whether the increase in transplant cases outweighed 
reduced patient stay due to better organs. The 
perfusion program would merely be the "tip of the 
iceberg" and its evaluation would only make sense 
in the larger context of the additional activities 
arising from its introduction. However, unless the 
changes in associated activities (which also have 
Significant costs and benefits) are clearly speci- 
fied, one can only "evaluate" a partial or sub- 
program by evaluating perfusion. 


These omissions, perhaps, arise out of the tradi- 
tional division of responsibility in hospitals: 
treatment and output generally being the responsi- 
bility of the medical staff. It appears that the 
administrator is merely informed by the medical 
staff that a program "need" exists - he is not 
expected to enquire too closely into the grounds 
for this’ opinion. In such a chlimaté, “evaluation is 
not possible. 


Subjunctive Rather than Probabilistic Quantitative 
Benefit Statements 


A number of assertions were made about benefits 
which: mrgGhts,)) could” ior “should *-foldiowttromithe 
program. No estimates were given as to their scale 
OL probability of occurrence. Hospitalization oF 
transplant cases could be reduced, but the amount, 
probability and total potential days involved were 
not stated. Transplant rejections could be re- 
Ggucedi but the «size of reduction, years Jof JiMitte 
saved, and quality of that life were not specified. 
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A program can be justified either in terms of cost 
reduction in other programs, or of improvement in 
mortality/morbidity outcomes, but these justifica- 
tions must be quantitative. Many good things might 
result from any new program, but they also might 
not. Health personnel, as a result of clinical 
training tend to think in anecdotal rather than 
statistical terms and to overemphasize the improb- 
able. But one cannot perform program evaluation 
without probabilistic, quantitative assessment of 
program outcomes. This is completely absent from 
Many proposals. Moreover, it must be remembered 
that such assessment must be comparative; it must 
analyze the differential impact of the program by 
examining hypothetical program/no program states of 
the world and specifying all other changes which 
would have to take place in response to the intro- 
duction of a program. None of these were described 
in the application being discussed, let alone 
quantified. 


An adequate evaluation is not possible, even if 
cost data is complete, if no quantitative estimates 
of expected benefit are made. These need not be 
one-point estimates or convertible into dollars 

but they must be quantified if the program is to be 
evaluated. Lying behind this, of course, are the 
conceptual problems of drawing an appropriate 
boundary around the program itself and of specify- 
ing expected "States-of-the-world" with and without 
the program. 


Prevention or Palliation of Environmental 
Damage-Asbestos 


Recent concern over chronic lung disease in asbes- 
tos workers represents awareness of the outcome of 
a program for dealing with worker health hazards, 
which is a conceptual parallel to screening or 
other types of preventive activity. The health 


hazard associated with asbestos has been known for 


a number of years. \From the point of view of the 
asbestos company, one might deal with this risk 
either by investing in equipment to protect workers 
against health damage or by paying workers more to 
induce them to undertake the risk. Either way, the 
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health hazard will be reflected in higher costs of 
production and, therefore, either higher asbestos 
prices on lower profits (or both)-: sithusi2s' fas ae 
cehouldsbea Enesnealth hazard, Sr areealy cose of 
production to society and should be reflected in 
the price society pays for its asbestos.3 


The choice of program will then be made by the 
company on the basis of relative costs. If the 
technology for lowering the risk is very expensive 
and hh ethe: cask: Ttselfivis rebatively,smalily or iGhe 
associated morbidity minimal, it may well be 
cheaper to pay workers extra to incur the risk. 


If workers are fully informed as to risk, this may 
be socially optimal -- dollars are balanced against 
risks all through society, most obviously in the 
transportation field. As individuals, each of us 
regularly incur certain risks to life and limb 
because we judge the costs of avoidance not worth 
it. As a society, we make the same judgements in, 
for example, highway spending. On the other hand, 
if the technology for risk reduction is cheap and 
the probability of morbidity or its severity are 
high, compenSation may be more costly than preven- 
tion. 


We may evaluate alternative programs for dealing 
with risk, as in the asbestos case, by investing in 
its reduction or compensating people for living 
with 12t; according to which is most ‘costly (always 
making the very powerful assumption that those who 
are compensated for bearing the risk are aware of 
its magnitude). The parallel between investing in 
risk-reduction through preventive measures such as 
maSs Screening, periodic check-ups or immunization, 
and the alternative strategy of dealing with mor- 
bidity Mac eltearisces.. is. close. 


This evaluation by the market or by comparison of 
compensation costs against prevention costs will 
break down when an outside group incurs some of the 
costs. From the points of view of the company and 
the workers, the evaluation is complete. Workers 
balance the extra pay against extra risk and accept 
or reject the risk. The company balances compensa-— 
tion against the cost of clean-up. 
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But from a social point of view, there may be 
hospital, medical, and other social maintenance 
costs arising from worker illness which are borne 
by the whole society through health and welfare 
programs. In this case, the balance struck between 
prevention and compensation by workers and company 
will underestimate the cost of the risk of illness 
because part of the cost is externalized -- 
incurred by neither -- and as a result, this 
balance will be biased. If, on the other hand, the 
governmental health insurance agency refuses to pay 
any benefits for illnesses generated by hazardous 
industrial environments, then the cost to the 
company of maintaining such environments rises. 
Workers have to be paid more either to provide 
health insurance coverage against the illness risk 
Or to compensate them for the risk of health care 
costs. jas: well, as the risk, of Iiiness pers Se... .ine 
evaluation of prevention versus compensation thus 
shifts to make prevention relatively cheaper. 


The general principle here is that evaluation of 
alternative strategies for dealing with a health 
hazard will depend on the perspective of the 
evaluator. In this case, the social perspective 
differs from that of worker/company, and evaluation 
made from the latter perspective may appear biased 
from the former .perspective. What is critical is 
that a complete specification of the impacts of the 
different approaches be made -- the compensation 
approach involves some costs to the wider community 
which may be avoided by prevention.4 What may be 
difficult is to find ways of ensuring that private 
decision-makers (workers or company) take account 
o£ such..costs:.in their internal evaluation. 


This difficulty is the more severe, of course, in 
that the company has an obvious incentive to under- 
state the magnitude of health hazards. This will 
either reduce the additional wages which workers 
would be paid to compensate them for accepting such 
risks, or dilute political pressures on the company 
to incur whagher «costs stor. maintaining safer enyvyi- 
ronments. With their knowledge of the process of 
production, the company is in a superior strategic 
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position. As a result, collection of information 
about and policing of hazardous environments may 
devolve upon governmental organizations and may 
require significant additional resources. Again ite 
should be stressed, however, that underestimation 
of health hazards does not lower the true social 
costs of a given production process. Apparent 
costs are lowered by transferring part of the total 
to workers, and/or to the broader society through 
the health care system. Beneficiaries of this 
transfer will be the owners of the company and/or 
the purchasers of the product. 


Day Care Surgery Program (DCU) 


Evaluation of a day care surgery program parallels 
fairly closely the discussion of home care in that 
it represents an alternative program intended to 
Substitute "for inpatient use. “Particularly vapplie-— 
able to the care of children, it enables one to 
avoid an inpatient episode altogether. As in the 
crude estimate of "Savings" from a home care 
program, one could estimate "Savings" from a day 
care unit by enumerating its patient load, multi- 
plying by the expected length of stay if all had 
been inpatients, and thus calculating total in- 
patient admissions and days avoided. lLess clear, 
of course, is by what such a total should be 
multiplied. The problem is that a DCU located in a 
hospital makes use of a wide range of hospital fa- 
cilities since for many purposes, the DCU patients 
can not be distinguished from inpatients (while 
actually undergoing surgery, for example). More- 
over, the types of cases that can be treated in a 
DCU will tend to be relatively straightforward, not 
involving highly complex diagnosis or treatment, 
and they must be compared with similarly straight- 
forward inpatient cases. It would obviously be 
incorrect to ‘calculate’ the total inpatient days 
avoided, as above, multiply this by the hospital 
per diem, and compare the total with the direct, 
1déntriied costs of running ‘the DCU itself? Wie 
procedure would overcount the costs of the in- 
patient cases and undercount the day care costs 


because of the large volume of hospital services 
used by such cases. 
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As in the home care case, the first problem of a 
proper evaluation is to determine a hypothetical 
hospital utilization pattern with or without the 
DCU. Since DCU cases represent discrete episodes, 
the problem of length of patient stay does not 
arise. Home Care augments an inpatient episode, 
day: Care: ‘SUDSULTUES, "for it. THowever,"1t is 
Critical to determine how many of the cases treated 
in a DCU would, in the absence of the DCU, have 
been treated as inpatients, as outpatients or not 
at all. Moreover, it is possible that a new unit 
will change admission patterns so that inpatient 
admissions actually rise due to physicians with 
multiple appointments shifting their admitting in 
response to the attraction of the DCU. 


Therefore, one has to build up two alternative 
views of the world, with and without DCU admission, 
taking into account the patient load patterns of 
all hospitals in the local catchment area, not 

just the hospital with a new unit. All the caveats 
in the home care program discussion about the 
likely response of admitting behaviour to the 
availability of new facilities apply equally in 
this case. 


Given these alternative utilization patterns, one 
must try “to Construct “associated costs. Buc here 
one encounters the problem of time horizon (which 
waS ignored in the discussion of home care). the 
costs associated with different utilization pat- 
terns will vary dramatically, depending upon 
whether one assumes that variation is small enough 
to be accommodated within a given physical 
Capacity, or whether one hypothesizes an expansion 
(or contraction) of scale of operations so as to 
keep utilization rates constant. For example, if a 
DCU actually does lower inpatient use, does one 
calculate only the variable cost savings associated 
with this reduction? These will be relatively 
small since all treatment costs are the same 
regardless of whether the case is inpatient or DCU. 
Or does one calculate a share of fixed cost - 
buildings and equipment - as well? 
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Neither one of these dominates the other -- they 
answer different questions. The variable cost 
approach is an answer to the short-run question - 
by how much might next year's inpatient budget 
actually. be. teduced if the Inpatient Toadstelie ine 
second addresses a more long-run policy question, 
if the hospital system adapts its capital stock 
(over time) to its new lower inpatient load, how 
much could be saved? Both are worth knowing, but 
the issue is somewhat obscured by the tendency of 
governments and non-profit institutions to do their 
capital accounting on a current basis. 


Capital should be charged for as it is used, with a 
charge combining an implicit rate of interest on 
invested capital and as realistic a depreciation 
rate as possible. Instead, current practice tends 
to charge capital expenditures to the year of 
occurrence, which undercounts the costs of ongoing 
programs and overcounts the costs of starting new 
ones. A related tendency is to treat existing 
capital as "free", making no charge for space or 
space maintenance when evaluating a new program 
merely because no new building is built. This 
latter procedure is valid only if, in the absence 
of the proposed program, the space and facilities 
it would occupy would stand heated, lighted, 
cleaned, and vacant -- a zero opportunity cost. 


The proper handling of capital costs is clearly 
important because any program which substitutes one 
institutional alternative for another is likely to 
affect capital requirements as well as operating 
costs. If more extensive use of DCU or home care 
facilities actually could reduce inpatient utiliza- 
tion (which, as stressed, is far from certain), 
then over a period of time the required stock of 
inpatient capital (relative to population) would 
fall, and the savings would be correspondingly 
greater. On the other hand, the payoff to new 
programs in terms of lowered capital needs may be 
deferredsit, the current capital ~stock is, already 
built up to the requirements of the old technology. 
a deferred payoff is, a reduced payort. 


Te 
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The development of operating cost differentials 
associated with different inpatient utilization 
patterns’ (with ior without DCU)itis hikely, toocbe 
messy in practice, but fairly straightforward in 
principle.5 pifficulties arise over the 

allocation of joint costs, but these can usually be 
dealt with by conventional allocation which will 
not biaS comparison between different modes of 
patient treatment in the same hospital. In 
general, one can partition direct and indirect 
operating costs into those incurred for DCU 
patients, those incurred for inpatients who are 
comparable to DCU patients, and those for services 
Or patients unrelated to the DCU. The per patient 
costs in the former two categories can then be used 
to build up the alternative cost patterns in the 
with DCU/without DCU care. The result will be a 
cost comparison from the perspective of the 
regional hospital system, not just from the 
hospital with a DCU. This can then be compared 
with the net difference in volume and type of 
utilization under the two systems. 


Shared Service Programs, Particularly Shared 


Administration Services 


Most program administrators are aware of the main 
advantage of shared service programs -- economies 
of scale. Frequently, this is presumed sufficient 
to overcome many of the non-financial difficulties 
inherent in shared service arrangements, including 
loss of autonomy, the risk involved in inflexibly 
tying all consumers into one producer (e.g., 
risking hospital operations through a strike in a 
central laundry) and standardization requirements. 
In other instances, non-financial considerations 
may be a sufficient incentive to use shared 
services, particularly where higher quality can be 
obtained by a group of organizations which would be 
beyond the reach of any one of them. Examples 
would include shared industrial engineering or bio- 
medical engineering programs, the interpretation of 
electrocardiograms by computers or by internists to 
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whom the tracings have been transmitted by tele- 
phone, improved laundry service when a central 
laundry replaces outdated facilities and shared 
data processing facilities. 


For shared administration programs, individual 
hospital organizations may benefit by obtaining the 
services of one or more highly qualified adminis- 
trators who would not otherwise be available to 
them. The problem then arises as to how to evalu- 
ate the services received. On the one hand, one may 
attempt to evaluate the savings involved over the 
costs of hiring equally or less qualified adminis- 
trators at the individual institutions. Indices of 
administrative costs per patient day might be 
compared. However, these ignore the effects of 
administration on the rest of the hospital. 


A second approach, to compare total costs of hos- 
pitals operated by shared administration programs, 
encounters difficulties arising from the lack of 
comparability between institutions and from the 
fact that many costs are beyond the control of the 
administrator. In any event, if either approach 
indicates coSt Savings, the question remains: to 
what end? 


If the emphasis iS on cost per unit, either admin- 
istrative or total, perspective is based on the 
short-range future, whereaS administrators are 
normally expected to have a longer perspective and 
to promote long-range planning for the institution. 
Conversely, if one looks to such indicators as 
number of complaints, staff turnover and frequency 
of untoward instances, costs are not directly 
reflected. But while these indicators reflect what 
may be considered organizational indicators of 
health, the perspective is again Short-range, far 
Shorter than normally expected of the administra- 
COV 


Other ratios often included are the percentage of 
institutional costs expended on administration, the 
infection rate, press notices, donations and volun- 
teer statistics. In some cases, evaluation of 
administrators is objective-oriented, taking into 
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account the completion of specific projects, cost 
Savings achievements, attainment of cost or produc-— 
tivity standards, maintenance of the hospital 
budget and utilization levels, and peer review. 


Similarly, accreditation reports, Organization 
audits and similar efforts to evaluate the 
Organization may only focus on a limited range of 
administrative activity. For instance, they may 
focus on systems of administration, plans for 
immediate activities, standing policies and pro- 
cedures, relevant by-laws and training programs, 
disaster plans, effective organizational arrange- 
ments and operations, and innovations. ; 


If, on the other hand, the job of administrator is 
defined as the long-term maintenance of an 
organization which is healthy, able to respond 
dynamically to current and future demands and to 
effect change where appropriate, then, in fact, no 
short-term measure will adequately determine 
administrative effectiveness. By corollary, with 
Such an arrangement (aS with the hiring of an 
individual administrator), one must accept the risk 
involved in extending the evaluation over a consid- 
erable period of time. Only in this way will the 
true effectiveness of the administrative orga- 
nization, individual or shared, be demonstrated. 


With that perspective, it is possible to compare 
alternative forms of organization, which may 
include hiring individual (though less well- 
qualified) unit administrators, and supporting them 
through consultative arrangements from organiza- 
tions such as hospital associations or consulting 
firs The obj ectives' of the sharedt servicer must 
be clearly stated in terms of effects, which will 
allow the comparison of alternative methods of 
service delivery. Furthermore, the time frame for 
evaluation must be appropriate, and this in itself 
will vary from service to service, in administra- 
tive as well as medical or professional matters. 
Economies of scale are not sufficient. The job 
itself must be done well. 
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Mass Vaccination Programs 


In April 1976, the Minister of National Health and 
Welfare announced a masSs vaccination program to 
protect Canadians against a possible pandemic of 
influenza. The program was to cover approximately 
ten million citizens and would have to be applied 
in avery short time period) if it was to; achieve 
any measure of success, i.e. preventing excessive 
deaths from pneumonia. The likelihood of such an 
epidemic arose from an epidemic of influenza, type 
A, which occurred in a garrison of the U.S. army, 
locatéd “at: Fort!) Dix) Nid.;. inowanuaryri9/64 5A5 new 
strain of human influenza A virus was then iso- 
lated. Serologic studies demonstrated that this 
virus was related anti-genically to the virus that 
is believed to have caused the influenza pandemic 
of 1918-19 and to the strain that has been circu- 
lating in swine since then. 


The Fort Dix episode represented the first 
documented human-to-human transmission of Swine 
influenza virus Since before 1930. "Influenza 
virus A/New Jersey/7/76, so-called swine influenza 
virus, represents a major change from the A/Hong 
Kong influenza viruses prevalent since 1968. (A 
current variant of these viruses, A/Victoria/75, 
waS epidemic in many parts of the world, including 
most of Canada in 1975-76.) Experience indicates 
that when a major antigen change occurs in preva- 
lent influenza A viruses, the new virus will 
rapidly spread world wide. This sequence of events 
waS particularly notable in 1957 and 1968 when the 
Asian and Hong Kong strains first appeared."(6) 


This program was to have mobilized a significant 
Share of the national health resources for its 
implementation. Setting aside problems related to 
the relative effectiveness of currently available 
vaccines and to the uncertain probability of the 
pandemic threatening the Canadian population, the 
program involved the following elements: 
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1. The general objective was to prevent the 
occurrence of an epidemic of influenza in the 
Canadian population, in order to reduce 
morbidity and excess mortality from that cause. 
In his statement to the media, the Minister 
referred to the estimated 20 million death toll 
of the pandemic of 1918-19. 
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(1) Program activities would include the 
identification and tegistration of all! "at 
risk" persons in the target population and 
(2) the administration of the vaccine. 


3. The program output would be the proportion of 
the target population vaccinated against the 
disease within the time span set for the 
program. 


4. The program outcome would be the decrease in 
morbidity and/or mortality which could 
reasonably be attributed to the immunization 
program itself rather than to other circum- 
Stances. In other words, it was expected that 
observed morbidity and/or mortality would be 
Significantly less than that expected in the 
absence of the program. 


As was implied earlier, the decision to organize 
‘this particular program occurred in a context of 
incomplete knowledge: the relative magnitude of the 
risk was unknown, and currently available vaccine, 
although of an efficacy proven to be about 70%, did 
not confer a long lasting protection. Hence, it 
was necessary to administer the vaccine as close as 
possible to the beginning of the high-risk period 
Of ‘the “epidemic: ' It'\was’ this ‘characteristic that 
created the difficulties in implementing the 
program for public health authorities. In spite of 
this, the decision to apply the program was deemed 
of greater value than a decision not to apply Lee 
The validity of >this! décision'(was*relative’ to the 
expected outcomes that had been identified as 
probable if the program had not been applied. In 
this example, mortality figures of 1918-19 were 
used by the media, most likely because of the 
antigenic similarity recognized earlier. Since 
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influenza epidemics occur periodically (17 have 
been documented in the U.S. since 1934)(7), one 
could question the choice of the 1918-19 pandemic 
for estimation purposes. Considering the changes 
that have affected so many Socioeconomic variables 
since 1918, it would seem that mortality rates 
observed in the 1957 epidemic would have been a 
better estimating standard than those of the 
earlier period. 


Since excess mortality in influenza epidemics 
affects mostly older age groups and patients with 
chronic illnesses, and Since an estimated 88% of 
the population aged 57 and over is considered to be 
already immunized against Swine influenza as a 
result of their contact with the wild virus between 
1919 and 1927,(8) there is a strong probability 
that the loss of "benefits" that would have 
occurred, had the program not been approved, was 
over-estimated. 


Perinatal Programs 


Infant mortality is the health status index that is 
most senSitive to the impact of medical and 
hospital services. Perinatal mortality is the 
component of infant mortality that accounts for 
most of infant deaths: hence, the great popularity 
of perinatal programs. 


1. The general objective of such programs is to 
reduce the perinatal mortality rate so that the 
infant mortality, rate wills, jover, a pervod of 
time, fall to that level which is deemed 
irreducible; that is, that level of mortality 
entirely explained by congenital abnormalities, 
etc. 


2. The program activities cover avast array 
of measures among which one may find one or 
several of the following: 


2.1 Pre- and post-natal classes made 
ava tlabhe to dali) ior, jaepr oportion, .of 
expecting mothers. 
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2.2 Changes in the methods of surveillance 
of pregnancy. 


2.3 Early referrals of at risk pregnancies 
from the general practitioner to 
specialists. 


2.4 A disincentive device to reduce the 
use of general anesthesia during 
delivery. 


2.5 Early resuscitation of the newborns. 


2.6 ‘IntenSive):care unvts efor hater tekvand 
diseased newborns. 


Program outputs in this example will be multiple 
and will vary with the activities intended to 
produce them. For instance, the proportion of 
expectant mothers who attend prenatal classes could 
be one variable. A better example would be the 
amount of change in the knowledge of those factors 
known to have an effect on the birth weight of the 
baby, such as cigarette smoking and diet. The 
amount of behavioral change, induced by this 
knowledge, would be an even more exact definition 
of the output of the first activity of the program. 


Timing of the first prenatal visit, subsequent 
prenatal visits and the professional activities 
included in each visit are other outputs of the 
program. The same may be said about changes in the 
proportion of high risk pregnancies followed up by 
specialists or reductions in the rate of general 
anesthesia and of surgical procedures during 
delivery. Outputs of improved resuscitation 
routines might be defined as the reduction of the 
mean period of apnea after birth or the improvement 
of the mean APGAR score. The reduction of 
mortality rates, standardized for birth weights, 
might serve aS a measure of the output of SOP a8 
for neonates. 
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The outcomes of this type of complex program may 
have different facets, according to one's point of 
view. The first may be a Significant reduction of 
perinatal mortality associated with the introduc-— 
tion of the program. The increase in Survivorship 
may be due to a reduction in the mortality of 
children born so handicapped that under normal 
circumstances they would have died. Therefore, the 
gain may not have been obtained through a reduction 
of the avoidable mortality (or undesirable 
mortality), but through a reduction of mortality 
among the most crippled newborns. Assuming that 
the aim iS maximization of survival in newborns 
without sequelae, evaluation of the program has to 
take this variable into account. 


In this perinatal program it may well be that the 
most rewarding activities happen to be the least 
expensive and the easiest to implement. They may 
also be those most acceptable to the clients of the 
program. When one considers the natural history of 
early perinatal mortality, which accounts for the 
largest part of the total perinatal mortality, one 
may find that after those behavioral changes that 
have to occur early in pregnancy, such as reduced 
tobacco use or increased physical exercise, proper 
risk classification is the next most efficient 
activity, if it leads to adequate surveillance of 
the pregnancy. In this example, it is possible 
that the most costly parts of the program produce 
the least desirable effects. 
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REFERENCES TO SECTION IV 


For an evaluation of a home care program which is 
not crude, see the study by M.L. Ingbar and S. Lee 
in S. Axelrod (ed.) The Economics of Health and 
Medical Care. Ann Arbor, Mich., 1964. 


The discrepancy .will, in general, be such that _the 
home care days are greater, not less, than the 
acute days avoided. This may occur for a number of 
reasons including the time and trouble for both 
patients and physicians in gaining access to two 
types of programs and the economic incentives to 
physicians to keep moving patients through acute 
care beds, (an incentive which does not apply in 
the case of home care). The motivational patterns 
may become quite complex, but they all point in the 
Same direction. 


From the company's point of view, of course, the 
optimal situation is for the health hazard not to 
be known or to be underestimated. In this case, 
workers will be under-compensated and the company's 
cost of production lowered. From a social point of 
view, this does not make asbestos cheaper to 


produce -- it merely hides some of the cost and 
places it on workers. 


Of course, if by forcing the company to compensate 
workers for health care costs as well, we encourage 
it to spend more on prevention, the result may be 
higher asbestos prices. (Although this may not 
necessarily apply -- asbestos mines may become 
worthless.) In this case, society trades lower 
health care costs for higher asbestos costs. But 
again, this is as it should be. Taxpayers who 
support health care programs should not subsidize 
asbestos users by enabling them to pay less than 
the full costs (including protection of worker 
health) of their products. 


For an example see R.G. Evans and G.C. Robinson. 
Economic Implications of a Day Care Surgery Unit 
(Mimeo) 

Vancouver, B.C., 1973. Report on NHG, Ottawa. 
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APPENDIX I 


Persons interviewed by members of the visiting team. 


A) Federal Government 
B) British Columbia 
C) Province of Quebec 


D) Saskatchewan 


A) Federal Government 


A. LeBlanc 

Director 

Operational Planning and Evaluation 
Social Service Programs Branch 
Department of National Health and Welfare 


N.E. Gayowsky 

Chief, Evaluation 

Fitness and Amateur Sport 

Department of National Health and Welfare 


André Gagnon 

Special Economic Advisor 
Policy, Planning and Evaluation 
Environment Canada 


C.T. Osborne 
Planning and Evaluation 
Energy, Mines and Resources 


B) British Columbia 


Mr. J.W. Mainguy 
Deputy Minister of Health 


Dr. Ke lets Denson 
Associate Deputy Minister 
Public Health Programmes 


Dr. As: Larsen 
Director of Epidemiology 


Mr D. Zink 
Director 
Division of Speech and Hearing 


ayes 


By). British columbia contc. Ga) 


Dr. F. McCombie 
Director 
Division of Dental Health Services 


Die aee GLay 
Assistant Director 
Division of Dental Health Services 


Mr. W.D. Burrowes 
Director of Vital. Statistics 


Miss Lavinia Crane 
Director 
Public Health Nursing 


Mr. P. Breel 
Senior Director 
Hospital Programs 


Mr. Alex Porteous 
Associate Deputy Minister 
Mental Health Programs 


Mr. Paul Pallan 
Development Group for Community 
Human Resources and Health Centres 


CG) Province of Quebec 
Ministé6ére des Affaires sociales du Québec 


M. Charles Chamard 
Sous-ministre adjoint 
Programmation 


M. Gilles Desrochers 
Sous-ministre adjoint 
Financement 


M. Marc Boucher 
Directeur 
Planification financiere 


M. Réjean Cantin 
Directeur 
Programmes de santé 
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C) Province of Quebec (cont'd) 
Ministére des Affaires sociales du Québec 


M. Henri-Paul Chaput 
Directeur 


Etablissements de services communautaires 


M. Claude Guimont 
Programmes de santé 


M. Jacques Privé 
Chef, Service des soins d'urgence 


Mme Nicole Martin 
Directeur 
Planification des services sociaux 


M. Claude Garcia 
Directeur 
Planif£ication. des services sociaux 


Dr. Roland Leblanc 
Directeur 
Inspection des é6tablissements de santé 


M. Roger Ladouceur 
Directeur 
Conception des normes 


D) Saskatchewan 


Mr. N. Duane Adams 
Assistant Deputy Minister 
Department of Health 


Dr. David Penman 
Chairman 
Medical Care Insurance Commission 


Mr. C.P. Feader 
Director Administrative Services 
Administrative Services Branch 
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D) Saskatchewan (cont'd) 


Mr. Steve Petz 

Executive Director 

Saskatchewan Prescription 
Drug Plan 


Mrs «dee Sanclarxr 
Assistant Executive Director 
Saskatchewan Hospital Services Plan 


Dr tebe eoULy, 
Director 
Health Promotion 
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APPENDIX 11 
WORKING PARTY ON PROGRAM EVALUATION 


TERMS OF REFERENCE 


To promote the evaluation of health programs, by: 


a. 


Dis 


C. 


extending the range of evaluation activities, 


improving the technical quality of evaluation 
projects and procedures, through encouraging 
the cumulative development of knowledge and 
the sharing of technical expertise, and 


encouraging the use of evaluation results; 


by providing: 


i 


A commentary on current practices in program 
evaluation, with recommendations and priorities 
as to how to improve them. 


Recommendations regarding directions for 
research into methodologies of and subject 
areas for evaluation, directed to priorities 
for funding (e.g. National Health Grants). 


Recommendations regarding how program 
evaluation results may be used, specific to 
existing and to new programs, and their use 
as planning inputs. 


Identified data requirements, and recommen- 
dations as to how to improve data 
availability. 


Information on those consultants or experts 
on program evaluation presentiy available. 


Recommendations regarding the dissemination 
of information in the field, (e.g. news- 
letters, national clearing house, etc.) 


Recommendations regarding the needed 
educational programs - full-time and 
continuing education. 
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